The present invention relates to technique of decoding a coded image, and more particularly relates to technique of detecting and eliminating coding noise derived from encoding.
Recently, the MPEG (Moving Picture Experts Group) standards are widely used to code a picture with high compression efficiency in the fields of broadcasting, communications and storage. According to the MPEG, an image is encoded by removing spatial and temporal redundancy.
In order to remove spatial redundancy, discrete cosine transform (hereinafter referred to as DCT) and quantization are employed. First, an image is divided into units each designated as a block including 8xc3x978 pixels, each block is converted through the DCT into coefficients of a frequency domain (hereinafter referred to as DCT coefficients), and the DCT coefficients are subjected to the quantization.
The quantization is an operation for dividing a DCT coefficient by both a quantizer matrix having a value corresponding to each frequency of a DCT region and a quantizer scale. Through the quantization, a value of a frequency component with a small DCT coefficient becomes zero. In general, energy is collected in a low frequency region in an image signal, and hence, a high frequency component is eliminated through this operation. However, the visual characteristic of a human being is poorer in a higher frequency region, and therefore, degradation in picture quality is less conspicuous as the quantizer scale used in the quantization is smaller (i.e., a step in the quantization is smaller).
In order to remove temporal redundancy, motion compensation is employed. In the motion compensation, an area closest to a macroblock in coding within a reference frame is selected with a macroblock of 16xc3x9716 pixels used as a unit, so as to encode difference values from the reference frame. When motion is not very fast, the difference values are substantially zero, and thus, temporal redundancy can be removed.
In general, when the bit rate of a code is high (i.e., the compression rate is low), the degradation in picture quality is very inconspicuous in the MPEG. When the bit rate is lower (i.e., the compression rate is higher), however, coding noise becomes more conspicuous and the picture quality is degraded. Typical coding noise occurring in the MPEG are blocking artifact (sometimes designated as block distortion or block noise) and ringing artifact (sometimes designated as ringing noise, corona noise or mosquito noise).
Blocking artifact is a phenomenon where a block boundary can be obviously seen in a tile-like shape. This noise is caused when an image signal within a block has merely low frequency components and the frequency component values are different from those of an adjacent block.
Ringing artifact is noise occurring in the vicinity of the edge of an image like a flying mosquito. This noise is caused because a high frequency component inherent to an image signal has been eliminated through the quantization.
Blocking artifact and ringing artifact are greatly remarkable types of degradation in picture quality unlike analog noises, and hence, several methods have been proposed for eliminating these coding noises.
An example of the method of eliminating blocking artifact is disclosed in Japanese Laid-Open Patent Publication No. 5-308623 (first conventional method). In this conventional method, a filter whose frequency characteristic is determined depending upon the maximum frequency of DCT coefficients and motion activity obtained based on a motion vector is applied.
Also, Japanese Laid-Open Patent Publication No. 7-177521 discloses a method in which a filter characteristic is determined by a quantizer scale and the size of a motion vector so that blocking artifact can be eliminated by applying a filter with the determined characteristic to a block boundary of a decoded image (second conventional method). In this conventional method, in the case where the quantizer scale is large and the motion vector is small, the filter characteristic is determined based on the quantizer scale alone. Alternatively, in the case where the quantizer scale is small and the motion vector is large, the filter characteristic is determined based on the size of the motion vector alone. Furthermore, in the case where both the quantizer scale and the size of the motion vector are medium, the filter characteristic is determined based on both of them.
On the other hand, an example of the method of eliminating ringing artifact is disclosed in Japanese Laid-Open Patent Publication No. 6-311499 (third conventional method). In this conventional method, an absolute sum of specific DCT coefficients of each block is obtained, so as to change the threshold value of a filter in accordance with the obtained absolute sum. Furthermore, in a frame that has been inter-frame coded, elimination of ringing artifact is carried out on a difference image, and then, reference frames where ringing artifact has been eliminated are added to the difference image so as to obtain a decoded image.
In general, occurrence of blocking artifact depends not on the maximum frequency of DCT coefficients but on the frequency component distribution of DCT coefficients and a difference in the frequency component distribution of DCT coefficients between adjacent blocks. When the filter characteristic is determined as in the first conventional method by using the high frequency component alone of the DCT coefficients, capability of eliminating blocking artifact is lowered.
The first conventional method further has the following problems: Since the filter characteristic is determined individually in the horizontal direction and the vertical direction, the operation required for the filter determination becomes numerous in volume. Furthermore, when the filter characteristic is determined based on DCT coefficients, capability of detecting blocking artifact is lowered in employing inter-frame coding (non-intra coding). In addition, in the case where the operation is controlled so that a strong filter is applied when the motion activity is large, no consideration is paid to the property of the reference frame. For example, in the case where motion compensation is accurately carried out even though the motion activity is large, a filter is still applied even if the pixel values in the difference image are all zero. As a result, the image can be degraded in the quality, for example, blurred. Accordingly, this method degrades the picture quality when applied to a non-intra coded image.
Moreover, even when the motion vector is large, a decoded image may include a high frequency component. Furthermore, with respect to a frame that has been inter-frame coded, even when the quantizer scale is large, a reference frame may include a high frequency component. In such a case, a decoded image may include a high frequency component. In this case, when a filter is applied to the decoded image as in the second conventional method, the picture quality is degraded.
Furthermore, the third conventional method has the following problems: Since the absolute sum of specific DCT coefficients of each block is obtained, calculation of the absolute sums of the DCT coefficients requires numerous processing. Moreover, since a decoded image is obtained by adding reference frames where ringing artifact has been eliminated, reference frames used in decoding are different from those used in coding. Accordingly, errors are accumulated on the decoded image, which makes the degradation in the picture quality conspicuous.
The present invention was devised to overcome the aforementioned conventional problems. An object of the present invention is providing a noise detecting method and a noise detector capable of definitely detecting a block boundary where blocking artifact occurs or a block where ringing artifact occurs so as to accurately detect such coding noise even in a non-intra coded block, and providing an image decoding apparatus capable of definitely eliminating coding noise with degradation in picture quality minimized.
Specifically, the noise detecting method of this invention includes the steps of: extracting coding information, including orthogonal transformation coefficients and a motion vector of each of multiple blocks of an image, from a bitstream that has been encoded through motion compensation of the image, orthogonal transformation and quantization, where the orthogonal transformation is performed on each said block; obtaining a reference area of each said block from a reference frame using the motion vector of the block; and detecting coding noise to be eliminated according to frequency component distributions of the orthogonal transformation coefficients of each said block and another one of the blocks within the reference frame overlapping the reference area of the block.
According to the noise detecting method of this invention, coding noise is detected according to not only the frequency component distribution of the orthogonal transformation coefficients obtained from a bitstream but also the frequency component distributions of the orthogonal transformation coefficients of blocks overlapping a reference area within a reference frame obtained by using the motion vector. Accordingly, coding noise can be definitely detected in a non-intra coded block and erroneous detection can be minimized.
In one embodiment the noise detecting method, blocking artifact is detected as the coding noise, and the coding noise detecting step includes the sub-steps of: classifying each said block into any of a plurality of classes according to the frequency component distribution of the orthogonal transformation coefficients of the block; obtaining new classes of a target one of the blocks and another one of the blocks, which is adjacent to the target block, according to classes of the target and adjacent blocks and classes of other ones of the blocks that overlap reference areas of the target and adjacent blocks within the reference frame; and detecting magnitude of blocking artifact occurring between the target and adjacent blocks according to the new classes.
In this embodiment, noise is detected according to not only the class of a target block for noise elimination but also the classes of the blocks overlapping the reference region within the reference frame obtained by using the motion vector. Accordingly, a block boundary where blocking artifact occurs can be definitely detected in a non-intra coded block. Furthermore, since the classification of the block is carried out not separately in the horizontal and vertical directions, blocking artifact can be detected with the processing operation reduced.
In another embodiment of the noise detecting method, ringing artifact is detected as the coding noise, and the coding noise detecting step includes the sub-steps of: classifying each said block into any of a plurality of classes according to the frequency component distribution of the orthogonal transformation coefficients of the block; obtaining a new class of a target one of the blocks according to a class of the target block and a class of another one of the blocks within the reference frame overlapping a reference area of the target block; and detecting magnitude of ringing artifact occurring in the target block according to the new class.
In this embodiment, noise is detected according to not only the class of the target block for noise elimination but also the classes of the blocks overlapping the reference area within the reference frame obtained by using the motion vector. Accordingly, a block where ringing artifact occurs can be definitely detected in a non-intra coded block. Furthermore, since the classification of a block is carried out not separately in the horizontal and vertical directions, ringing artifact can be detected with the processing operation reduced.
In still another embodiment of the noise detecting method, blocking artifact is detected as the coding noise, and the coding noise detecting step includes the sub-steps of: extracting DC components of the orthogonal transformation coefficients of a target one of the blocks and another one of the blocks, which is adjacent to the target block, from the coding information; and detecting magnitude of blocking artifact occurring between the target and adjacent blocks according to not only the frequency component distribution of the orthogonal transformation coefficients but also an absolute value of a difference in the DC component between the target and adjacent blocks.
In this embodiment, the absolute value of a difference in the DC component between the target and adjacent blocks is checked, so the blocking artifact can be detected very accurately.
In still another embodiment of the noise detecting method, the coding noise detecting step includes the sub-steps of: extracting a quantizer scale of the target block from the coding information; and detecting magnitude of blocking artifact occurring between the target and adjacent blocks according to not only the frequency component distribution of the orthogonal transformation coefficients but also the absolute value of the difference in the DC component and the quantizer scale.
In this embodiment, the absolute value of a difference in the DC component between the target and adjacent blocks is checked in accordance with the quantizer scale of the target block. Accordingly, blocking artifact can be detected very accurately.
In still another embodiment of the noise detecting method, the coding noise to be eliminated is detected according to not only the frequency component distribution of the orthogonal transformation coefficients but also a size of the motion vector of a target one of the blocks or another one of the blocks that is adjacent to the target block.
Since this embodiment uses the size of the motion vector, noise can be appropriately detected. Because, in general, the larger the size of a motion vector is, the greater the magnitude of coding noise is.
In still another embodiment of the noise detecting method, blocking artifact is detected as the coding noise, and the coding noise detecting step includes the sub-steps of: extracting DC components of the orthogonal transformation coefficients of the target and adjacent blocks from the coding information; and detecting magnitude of blocking artifact occurring between the target and adjacent blocks according to not only the frequency component distribution of the orthogonal transformation coefficients and the size of the motion vector but also an absolute value of a difference in the DC component between the target and adjacent blocks.
In this embodiment, the absolute value of a difference in the DC component between the target and adjacent blocks is checked, so the blocking artifact can be detected very accurately.
In still another embodiment of the noise detecting method, the coding noise detecting step includes the sub-steps of: extracting a quantizer scale of the target block from the coding information; and detecting magnitude of blocking artifact occurring between the target and adjacent blocks according to not only the frequency component distribution of the orthogonal transformation coefficients and the size of the motion vector but also the absolute value of the difference in the DC component and the quantizer scale.
In this embodiment, the absolute value of a difference in the DC component between the target and adjacent blocks is checked in accordance with the quantizer scale of the target block. Accordingly, blocking artifact can be detected very accurately.
In still another embodiment of the noise detecting method, blocking artifact and ringing artifact are detected as the coding noise, and one of blocking artifact and ringing artifact is chosen for each said block as the coding noise to be eliminated according to magnitude of the blocking artifact and the ringing artifact.
In this embodiment, one of blocking artifact and ringing artifact is chosen for each said block. Accordingly, the processing operation and a memory capacity required for the noise elimination can be reduced. As a result, blocking artifact and ringing artifact can be both eliminated appropriately.
In still another embodiment of the noise detecting method, the coding noise detecting step is carried out on each of multiple fields in an interlaced image.
In this embodiment, coding noise can also be detected very accurately from an interlaced image.
The noise detector of this invention includes means for receiving coding information as an input, detecting coding noise to be eliminated and outputting a result of the detection. The coding information includes orthogonal transformation coefficients and a motion vector of each of the multiple blocks of an image. The coefficients and vector are obtained from a bitstream that has been encoded through motion compensation of the image, orthogonal transformation and quantization. The orthogonal transformation is performed on each said block. The coding noise is detected according to frequency component distributions of the orthogonal transformation coefficients of each said block and another one of the blocks within a reference frame overlapping a reference area of the block. And the reference area is obtained from the reference frame using the motion vector of each said block.
The noise detector of the invention detects coding noise according to not only the frequency component distribution of the orthogonal transformation coefficients obtained from a bitstream but also the frequency component distributions of the orthogonal transformation coefficients of blocks overlapping a reference area within a reference frame obtained by using a motion vector. Accordingly, coding noise can be definitely detected in a non-intra coded block and erroneous detection can be minimized.
The image decoding apparatus of this invention includes: the aforementioned noise detector; a decoding unit for decoding the bitstream and outputting the coding information including the orthogonal transformation coefficients of each said block and a motion vector of a target one of the blocks; and a coding noise eliminating unit for eliminating the coding noise based on the detection result output by the noise detector.
In the image decoding apparatus of this invention, coding noise can be definitely detected in a non-intra coded block, and degradation in picture quality derived from elimination of coding error detected erroneously can be minimized.
In an embodiment of the image decoding apparatus, the noise detector detects ringing artifact as the coding noise and extracts a quantizer scale of the target block from the coding information, and the coding noise eliminating unit uses a value in accordance with the quantizer scale as a threshold value for detecting an edge pixel not used in eliminating the ringing artifact.
In this embodiment, pixels used in eliminating ringing artifact can be selected in accordance with the quantizer scale. Accordingly, ringing artifact can be definitely eliminated with the degradation in picture quality at the edges minimized.